Skip to content

🐛 Refactored retry logic to include failing case in AWS master #7809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 4, 2025
Merged

🐛 Refactored retry logic to include failing case in AWS master #7809

merged 2 commits into from
Jun 4, 2025

Conversation

@GitHK
Copy link
Contributor

@GitHK GitHK commented Jun 4, 2025 •
edited
Loading

What do these changes do?

When aiohttp.client_exceptions.ClientPayloadError would be raised due to a aiohttp.http_exceptions.ContentLengthError it will simply be reported as an error and no retry would be applied.

This refectories the retrial to also include this case.

Traceback (most recent call last):
  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/client_proto.py"", line 135, in connection_lost
    uncompleted = self._parser.feed_eof()
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File ""aiohttp/_http_parser.pyx"", line 511, in aiohttp._http_parser.HttpParser.feed_eof
aiohttp.http_exceptions.ContentLengthError: 400, message:
  Not enough data to satisfy content length header.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_sdk/node_ports_common/file_io_utils.py"", line 226, in download_link_to_file
    await _file_chunk_writer(
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_sdk/node_ports_common/file_io_utils.py"", line 157, in _file_chunk_writer
    while chunk := await response.content.read(CHUNK_SIZE):
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/aiohttp/streams.py"", line 392, in read
    raise self._exception
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <ContentLengthError: 400, message='Not enough data to satisfy content length header.'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ""/home/scu/.venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py"", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py"", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/fastapi/applications.py"", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/applications.py"", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py"", line 187, in __call__
    raise exc
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py"", line 165, in __call__
    await self.app(scope, receive, _send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/opentelemetry/instrumentation/asgi/__init__.py"", line 743, in __call__
    await self.app(scope, otel_receive, otel_send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py"", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py"", line 53, in wrapped_app
    raise exc
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py"", line 42, in wrapped_app
    await app(scope, receive, sender)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py"", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py"", line 734, in app
    await route.handle(scope, receive, send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py"", line 288, in handle
    await self.app(scope, receive, send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py"", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py"", line 53, in wrapped_app
    raise exc
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/_exception_handler.py"", line 42, in wrapped_app
    await app(scope, receive, sender)
  File ""/home/scu/.venv/lib/python3.11/site-packages/starlette/routing.py"", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/fastapi/routing.py"", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/fastapi/routing.py"", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/fastapi/requests_decorators.py"", line 94, in wrapper
    return await handler_task
           ^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/long_running_tasks/_task.py"", line 262, in _cancel_asyncio_task
    await asyncio.wait_for(
  File ""/usr/local/lib/python3.11/asyncio/tasks.py"", line 489, in wait_for
    return fut.result()
           ^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/servicelib/long_running_tasks/_task.py"", line 31, in _await_task
    await task
simcore_sdk.node_ports_common.exceptions.TransferError: Error while transferring to/from [REDACTED]

Related issue/s

How to test

Dev-ops

@GitHK GitHK self-assigned this Jun 4, 2025
@GitHK GitHK added this to the Bazinga! milestone Jun 4, 2025
@codecov
Copy link

codecov bot commented Jun 4, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 60.00000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 86.34%. Comparing base (6748f5e) to head (6cb5606).
Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #7809      +/-   ##
==========================================
+ Coverage   85.40%   86.34%   +0.93%     
==========================================
  Files        1850     1257     -593     
  Lines       71910    53935   -17975     
  Branches     1218      176    -1042     
==========================================
- Hits        61414    46568   -14846     
+ Misses      10153     7309    -2844     
+ Partials      343       58     -285
Flag Coverage Δ
integrationtests 64.25% <60.00%> (+0.09%) ⬆️
unittests 88.31% <6.66%> (+1.60%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.09% <60.00%> (+0.23%) ⬆️
agent 96.29% <ø> (ø)
api_server 91.73% <ø> (ø)
autoscaling 96.03% <ø> (ø)
catalog 92.29% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 91.79% <ø> (ø)
datcore_adapter 97.94% <ø> (ø)
director 76.73% <ø> (ø)
director_v2 77.69% <ø> (+0.12%) ⬆️
dynamic_scheduler 96.69% <ø> (ø)
dynamic_sidecar 90.08% <ø> (ø)
efs_guardian 89.65% <ø> (ø)
invitations 93.00% <ø> (ø)
payments 92.57% <ø> (ø)
resource_usage_tracker 88.98% <ø> (ø)
storage 87.53% <ø> (+0.07%) ⬆️
webclient ∅ <ø> (∅)
webserver 83.74% <ø> (-0.02%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6748f5e...6cb5606. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@GitHK GitHK marked this pull request as ready for review June 4, 2025 12:24
@GitHK GitHK changed the title refactored retry logic Refactored retry logic to include failing case in AWS master Jun 4, 2025
@GitHK GitHK changed the title Refactored retry logic to include failing case in AWS master 🐛 Refactored retry logic to include failing case in AWS master Jun 4, 2025
@pcrespov pcrespov self-requested a review June 4, 2025 12:30
@sonarqubecloud
Copy link

sonarqubecloud bot commented Jun 4, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@pcrespov pcrespov merged commit 87fc3e7 into ITISFoundation:master Jun 4, 2025
85 of 91 checks passed
@GitHK GitHK deleted the pr-osparc-try-fix-issue-with-ports branch June 4, 2025 12:56
GitHK pushed a commit to GitHK/osparc-simcore-forked that referenced this pull request Jun 4, 2025
Revert "🐛 Refactored retry logic to include failing case in AWS master (
5d30806 
ITISFoundation#7809)"

This reverts commit 87fc3e7.
@GitHK GitHK mentioned this pull request Jun 4, 2025
Merged
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Jun 6, 2025
Closed
92 tasks
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Aug 5, 2025
Closed
88 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sanderegg sanderegg Awaiting requested review from sanderegg

@matusdrobuliak66 matusdrobuliak66 Awaiting requested review from matusdrobuliak66

@bisgaard-itis bisgaard-itis Awaiting requested review from bisgaard-itis

@pcrespov pcrespov Awaiting requested review from pcrespov

Assignees

@GitHK GitHK

Labels

None yet

Projects

None yet

Milestone

Bazinga!

Development

Successfully merging this pull request may close these issues.

2 participants

@GitHK @pcrespov